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ABSTRACT 

The vision of this project is to develop lane and object detection in 
Autonomous Vehicle system to run efficiently in normal road condition and to 
eliminate the use of high cost Light based LiDAR system to implement high 
resolution cameras with advanced computer vision and deep learning 
technology to provide an Advanced Driver Assistance System (ADAS). 
Detecting lane lines could be a crucial task for any self-driving autonomous 
vehicle. Hence, this project was focused to identify lane lines on the road using 
OpenCV. The OpenCV tools such as colour selection, the region of interest 
selection, grey scaling, canny edge detection and perspective transformation 
are being employed. This project is modelled as an integration of two systems 
to solve the real-time implementation problem in autonomous vehicles. The 
first part of the system is lane detection by advanced computer vision 
techniques to detect the lane lines to command the vehicle to stay inside the 
lane marking. The second part of the system is object detection and tracking is 
to detect and track the vehicle and pedestrians on the road to get a clear 
understanding of the environment to plan and generate a trajectory to 
navigate the autonomous vehicle safely to its destination without any crashes, 
this is done by a special deep learning method called transfer learning with 


Single Shot multibox Detection (SSD) algorithm and Mobile Net architecture. 
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INTRODUCTION 

An autonomous vehicle is one that can drive itself from a 
Starting point to a predetermined destination in “autopilot” 
mode using various in-vehicle technologies and sensors, 
including adaptive cruise control, active steering (steer by 
wire), anti-lock braking systems (brake by wire), GPS 
navigation technology, lasers and radar. It utilises a fully 
automated driving system in order to allow the vehicle to 
respond to external conditions that a human driver would 
manage. It transform transportation into a utility available to 
anyone, anytime. This requires advances in many aspects of 
vehicle autonomy, ranging from vehicle design to control, 
perception, planning, coordination, and human interaction. 
Computer Vision gives vision to the AV and make the vehicle 
to perceive through the world by understanding the objects 
in the environment to make live critical intelligent decision 
by the use of advanced Computer Vision techniques, Machine 
Learning and Deep Learning [2]. 


A. LANE DETECTION 

An increasing safety and reducing road accidents, thereby 
saving lives are one of great interest in the context of 
Advanced Driver Assistance Systems (ADAS). Apparently, 
among the complex and challenging tasks of future road 
vehicles is road lane detection or road boundaries detection. 
It is based on lane detection (which includes the localization 
of the road, the determination of the relative position between 
vehicle and road, and the analysis of the vehicle’s heading 
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direction). One of the principal approaches to detect road 
boundaries and lanes using vision system on the vehicle. 
However, lane detection is a difficult problem because of the 
varying road conditions that one can encounter while driving. 
In this paper, a vision-based lane detection approach capable 
of reaching real time operation with robustness to lighting 
change and shadows is presented. The system acquires the 
front view using a camera mounted on the vehicle then 
applying few processes in order to detect the lanes [6]. 


B. OBJECT DETECTION USING TRANSFER LEARNING 
Transfer learning generally refers to a process where a model 
trained on one problem is used in some way on a second 
related problem. 


In deep learning, transfer learning is a technique whereby a 
neural network model is first trained ona problem similar to 
the problem that is being solved. One or more layers from the 
trained model are then used in a new model trained on the 
problem of interest. 


We used Transfer Learning method to implement this 
convolutional neural network evaluated on the PASCAL VOC 
detection dataset. This network has 24convolutional layers 
followed by 2 fully connected layers. Instead of the inception 
modules used by SSD- MobileNet architecture simply use 1*1 
reduction layers followed by 3*3 convolutional layers, Single 
Shot object Detection (SSD) can achieve a single crop top-5 
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accuracy of 88% on the ImageNet. ssd final layer predicts 
both class probabilities and bounding box coordinates. This 
model is called the MobileNet [1]. The Architecture of 
MobileNet is shown in the figure 1. 
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Figure 1: SSD-MobileNet architecture (source: SSD: 
Single Shot MultiBox Detector, unified real-time object 
detection) 


I. ALGORITHM 
The block diagram of the proposed lane detection system is 
shown in figure 2. 
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Figure 2: block diagram of lane detection system 
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The description of each block is explained below: 

1. Camera input: the HD cameras that are placed in the 
front of the autonomous vehicle is used as the input 
source for finding the lane and object detection process. 

2. Video capturing: the real time video is captured by the 
camera and is taken as the input. 

3. Frames: the captured video is split into frames for 
further processing. 

4. Computer vision and image processing: this process is to 
perform functions like enhancing the picture quality and 
smoothing the images. 

5. Background subtraction: to get the undistorted images 
we use techniques like Gaussian blur, smoothing, and 
edge detection to the input images. 

6. Apply algorithms :the algorithms Namely canny edge 
detector, perspective transformation and _ sliding 
window techniques are used to get the clear detection of 
lanes. 

7. Lane detecting zones: after applying the algorithms the 
ROI is found and the Hough transform is applied to the 
cropped image to detect the lines in the image. 

8. Output: here the lane lines are programmed to detect 
lanes even without the lane markings along with the 
road map for advanced driver assistance system (ADAS). 


The advanced techniques used are canny edge detection, 
perspective transformation, sliding windows. 


II. DESIGN 

The step by step lane detection pipeline is shown in the figure 
3, it outputs the stacked images of lane line in each process 
from distortion correction to perspective transform. 
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Figure 3: lane detection pipeline 


A. DISTORTION CORRECTION 

By calibrating a camera, we can enable a self-driving car to 
know the location of pedestrians, dogs, and other objects 
relative to the car [4]. 


Print Image 





Calibrating the camera is the process of using a known real- 
world pattern (e.g. a chessboard) to estimate the extrinsic 
parameters (rotation and translation vectors) and intrinsic 
parameters (e.g. focal length, optical centre, etc.) of a 
camera’s lens and image sensor to reduce distortion error 
caused by the camera’s imperfections as shown in figure 4. 


Distorted image Undistorted image 








Figure 4: effect of calibration on the distorted image 


B. EDGE DETECTION 

The first step in detecting this marking is converting the 
colour-segmented image grayscale and then using edge 
detection [3]. In this we have used canny edge detector to 
produce the best edge image as shown in figure 5. 








Figure 5 result of canny edge detection 
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C. PERSPECTIVE TRANSFORMATION 

Certain features like the curvature of the road lane markings 
is found using a technique called perspective transformation 
that transforms the 


undistorted image to a "birds-eye view’ of the road as shown 
in the figure 6 , this focuses only on the lane lines and 
displays them in such a way that they appear to be relatively 
parallel to each other. 





Fig 6 perspective transformation of the original image 


D. SLIDING WINDOW SEARCH 

Now the lanes can be identified by using the sliding window 
search approach to identify the regions of the frame with the 
highest density of non-zero pixels as shown in the figure 
7.we have used 9 sliding windows for better results [13]. 
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Fig 7 sliding window search 


Il. RESULTS 

A. LANE DETECTION 

The final result of the lane detection is shown in the figure 8, 
figure 9 and figure 10, it outputs the area inside the detected 
lane marking in green colour to generate the driveable 
trajectory to send the control commends to the actuator. Our 
project output has been tested on three real time captured 
input videos which also describes the information that 
contains the lane info along with road curvature, deviation 
and the road map. 
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Fig 9 Detected lane on the challenge video 
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Fig 10 detected lane on the harder challenge video 


B. OBJECT DETECTION AND TRACKING 

The output from the object detection and tracking is shown 
in the figure 11, it detect and tracks the object in the road 
with bounding box with the accuracy of 85%. The use of SSD 
and Mobile Net gives faster detection and tracking with 
better accuracy of the model as shown in figure 12. 











Fig 11 Detected and Tracked vehicle 
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This model shows 98% accuracy in training and 83% in 
validation as shown in figure 12. 


The reason for the drop of accuracy in validation set is that 

the model over fitted with the training data. In real- time 

dropout layers can be added to the model to prevent this 

overfitting [1]. 
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Fig 12 Accuracy and Loss of Transfer Learning Model 


CONCLUSIONS 

The autonomous vehicle system is developed to run the 
Vehicle precisely with advanced computer vision and deep 
learning techniques. The output from the lane detection 
system is used to generate reference trajectory to keep the 
autonomous vehicle inside the lane to avoid accidents by 
advanced computer vision. The object detection and tracking 
with the help of Single Shot object Detection (SSD) algorithm 
gives very fast and accurate results to detect and track the 
objects in the road at high frame rate. The accuracy and 
performance of this network are greater to that human in 
making optimized and rapid decisions, this results in reduced 
accidents causing random human error. In lane detection, 
previous research has been focusing on complex algorithms 
and expensive sensing devices to process input data. These 
kinds of methods usually involve heavy computing and high 
cost experiments. This project pursues an alternative way in 
lane detection. This project was mainly focused on reducing 
the cost of sensors and to use simple and efficient algorithm 
to detect moving objects and lane from just the camera input. 
This method will be efficient in identifying the lane lines and 
remove interfering noises in lane detection. 
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